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3 ' Let G be a classical real group, let K he a maximal compact subgroup in G. 

There exists a canonical embedding of a symmetric space G/K to some non- 
compact hermitian symmetric space G° /K° such that dims G/K = dime G° /K° ^ 
(see Addendum A). Consider a problem about restriction of a highest weight 
representation of the group G° to the subgroup G. We use the term "kernel- 
^H I representations" (by analogy with Bergman kernel-function) for restrictions of 

P^ ■ this type. 

A studying of kernel-representations was initiated by F. A. Berezin [0 shortly 
before his death (proofs were published only in 1994 by Upmeier and Unter- 



berger |31 ). In last years an interest to this subject appeared again. Partially 
it is related to continuation of Berezin work, partially this interest has inde- 
pendent reasons (|ll, Q, H, H,!, llll, 0, [H, H)- It appeared that 



spectra of kernel representations are very intricate. For instance (see [ pO[ ) they 
^ I include all representations which can appear in spectra of Howe dual pairs; a 

J;0 . decomposition of Howe dual pairs is known as extremely rich spectral problem. 

G.I.Olshanskii and myself used kernel-representations for construction of exotic 

unitary representations of the groups 0(p, q), U(p, q), Sp(p, q), see p3[ . 

f~^ I Emphasis that in the case of spaces of scalar- valued functions (see §1) a limit 

0^ ' of kernel representations as a parameter a tends -l-oo is the usual representation 

On ' of the group G in the space L'^{G/K), i.e. the kernel-representations can be 

r~| ■ considered as natural " deformation" of the spaces L^ {G/K'^ 

The main purpose of this paper is a decomposition of kernel-representations 
to "blocks" having more or less uniform spectra ( in fact we consider only the 
case G = 0{p,q); the cases G = U(p, g), Sp(p, q) are similar and all information 
necessary for repetition of our construction for \J{p,q), Sp{p,q) is contained in 
the paper). The idea to construct decompositions of this type in various prob- 
lems of harmonic analysis is old (see Gelfand-Gindikin H). G.I.Olshanskii [Eq| 
C^ ' proposed a way (noncommutative Hardy spaces) for separation of highest weight 

increments (see survey on this subject). There are also papers of Gindikin 
and Molchanov fl^] containing solution of the problem for hyperboloids. Our 
method is based on theorems about restrictions of discontinuous functions to 
submanifolds. In |lj, ||2j, [g3|, |3|,Q this method was used for separation of 
the discrete part of spectrum) . 

^ Partially supported by grant RFBR 98-01-00303 and Russian program of support of 
scientific scliools (grant RFBR 96-01-96249). 

•^In a vector- valued case, a kernel-representation can be obtained by a deformation of a 
space of L^-sections of a vector bundle over G/K). 



In §1 we define kernel representations for groups 0{p,q) (in the spaces 
of scalar- valued functions). In §2 we evaluate some integrals over orthogonal 
groups, which seems pleasant itself. In §3 we prove existence of restriction op- 
erators. In §4 we prove similar statements for general kernel representations (in 
spaces of vector- valued functions). 

Addendum A contains brief discussion of definition of kernel representations 

Addendum B is devoted to analytical continuation of Planchcrcl formula for 
kernel representations. 

I thanks G.I.Olshanskii, V.F.Molchanov, and B.Orstcd for numerous discus- 
sion of this subject. I also thanks H.SchlichtruU and G. van Dijk for discussions 
and references. 



§1. Kernel-representations 



1.0. Positive definite kernels. Let A* be a set. Recall that a function 
K(x, y) on X X X is called by positive definite kernel if for all xi, . . . ,x„ £ X 
the matrix 

K[xx,Xy) ... K[xx,Xn) 
K{Xn,Xi) ... K{Xn,Xn) 

is positive definite. A positive definite kernel defines an unique (up to a natural 
equivalence) Hilbert space H and a system of vectors 6^ (z H [supercomplete 
basis or system of coherent states), enumerated by points x Cz X such that 

1- {Ox,Oy)H = K{x,y) for aU y,x e X 

2. The linear span of vectors 9x is dense in H. 

For each vector v Cz H wc define a function fv{x) on X by the rule 

/^,(a;) = {v,dx)H 

We denote by H° the space of all function obtained in this way. 

1.1. Matrix balls. Let p ^ q. By Bpq we denote the space oi p x q 
matrices with norm < 1 over R (a norm of a matrix is the norm of operator in 
Euclidean space). By Bp^q we denote the closure of Bp^q in M.^'', i.e. the set of 
matrices with norm ^ 1. 

By 0(p, q) we denote a group of real (p + q) x (p + q) matrices g = ( , 

preserving bilinear form with the matrix i )• ^^'^ group 0{p,q) acts 

on the matrix ball Bp^^ (and on Bpq) by fractional linear transformations 

ZH^zIsl := {a + zc)-\b + zd) 



It is easy to check that the action of 0{p,q) on Bp^^ is transitive, and the 

stabihzer of the point z = consists of matrices r, w ) '^here a e 0{p),d G 

0{q). Hence 

Bp,,~0(p,g)/0(p)xO(g) 

By M/i we denote the space of matrices z e Bp.g such that rank of (1 — zz*) 
is h. Obviously, the sets M^ are orbits of the group 0(p, q) on the boundary of 
the matrix ball Bp^g . By M^, we denote the closure of the orbit M/j : 

1.2. Berezin spaces. Let 

a = 0, 1, 2, . . . ,p — 1 or a>p—l 

The Berezin kernel Kq, — Ka{z,u) is the function on B^^ x Bpq given by the 
formula 

Ka{z, u) = det(l - zu*)-°' = det(l - uz*)~" 

By well-known Berezin theorem M for a indicated above the Berezin kernel is 
positive definit^ Hence we obtain the space of functions on the matrix ball. 
Let us describe this space. 

By I?'(Bp q) we denote the space of distributions supported in the open 
matrix ball Bpg. Let us define the scalar product in I?'(Bp^g) by the formula 

(0, V)a := {(b{z)^j{u),det{l - ZM*)-"} 

The Berezin space Ha = Ha(Bpq) is the completion of pre-Hilbert space V^Bpq) 
(in our case, a super-complete basis 6^ consists of (5- functions). 

1.3. Dual realization of Berezin spaces and kernel-representations. 

By Sa we denote the delta-function supported at the point a G Bp g. To each 
element v G Ha{Bp.q) we assign the function f^ on Bp^g defined by the formula 

Hence we obtain some space _ff°(Bp.g) ~ Ha{Bp^q) consisting of functions on 

Bp^q 

It is readily seen that elements of the space H°(Bpq) are real analytical 
functions on Bp g. Then we define the unitary representation of the group 
0{p,q) in iJ°(Bp^g) by the formula 

^"(c J )/W = (« + ^c)-"/((a + zc)-i(6 + zd)) (1.1) 



•^There were also papers of Gindikin(1975), Vergne— Rossi(1976) and Wallach(1979) on this 
subject, see references in |23| or [p|. 



We name the representations Ta by term kernel-representations 

Remark. Berezin H in the case of hermitian symmetric spaces used the 
term canonical representation. He gave this term from the paper iz p3]. This 
term can have many different senses ; in fact A.M.Vershik, I.M.Gelfand, M.I.Graev 
themself in paper [^ and subsequent works use the term "canonical representa- 
tion" for multiphcative integrals and and infinitely divisible representations. In 
fact the class of infinitely divisible representations have very small intersection 
with the class of "kernel-representations", this small intersection is a chance. 

1.4. Holomorphic continuation. By Bp ,j(C) we denote the set of all 
complex p X q matrices with norm < 1. The kernel Ka extends to the function 
det(l — zu*)~°' on Bp.g(C) x Bp_g(C), which is holomorphic in z and antiholo- 
morphic in u; this function is positive definite on Jip^q{C) (|l[). Then we deduce 
that functions / S H°{Bp^q) extend to holomorphic functions on Bp^g(C). 



§2. Some integrals over orthogonal groups 



Consider a matrix A. By [A]p we denote its left upper corner having a size 
p X p. We denote the Haar measure on SO(n) by dan{g)- 

2.1. The maps T™. Let us wright a matrix g E SO(n) as a (m + (n — 

m)) X [m + (n — m)) block matrix „ rp ] ■ Consider the map 

^"- ( i? T ) ^T-R{l + P)-^Q 

defined almost everywhere 

Proposition 2.1. a) T™ is a map SO(n) to SO(n - m). 

'^' 5+T *"^P"6S {o\p= T"-P(g) + 1 

where {S}p denotes the right lower p x p corner of the matrix S. 

Remark. The map T is a value of Livshic characteristic function x(z) = 
T + zR{l - zP)-^Q at the point z = -1 (see |l|). 

Proof. Statement c) can be easily checked by direct calculation using 
Frobenius formula for inversion of block matrix (see ||7|, § II. 4) , 

A B y^ _ ( A-^+ A-^B{D - CA-^B)-^CA-^ -A-^B{D - CA-^B)-^ 
CD) ~\ -{D-CA-^B)-^CA-^ (D-CA-^B)-^ 

Statements a),b) are consequences of c). 
Proposition 2.2. Let A,B,eSO(n- m). Then 



1 \ / 1 
A r\ B 



AT"'{g)B 



Proof is obvious. 

2.2. Another proof of Proposition 2.1. Consider the space Vn := 

R" © R" equipped with indefinite bilinear form 

n n 

Ln{ix,y),{x',y')) -.^^Xkx'f, -^yky'k 
fc=i k=i 

A graph of an unitary operator R" -^ R" is a L„-isotropic subspace in Vn = 
R"®R". 

For each Hnear relation M : Vn ^ Vn-m (recall that a linear relation is a 
subspace M C Vn (B Vn-m, for details see Q) and each subspace H C Vn it 
is possible to define a subspace MH C Vn-m in the following way. A vector 
y e Vn-m is an element of MH iff there exists x ^ H such that (a;, y) G M . 

Consider the linear relation Z^~"^ : Vn =^ Vn-m consisting of all vectors 
having the form 

{{{v,h),{w,~h)},{v,w}) e {M"eR"}e{M"-"eR"-"} 

The condition {u,v),{u',v') £ Z^^™ imphes Ln{u,u') — Ln-m{v,v'). Hence 
the transformation H h^ Z"^™_ff maps isotropic subspaces to isotropic sub- 
spaces. 

It is readily seen that the map T™ coincides with the transformation H i-^ 
Z^~"^H of isotropic grassmannians. Now Proposition 2.1 becomes obvious. 

2.3. A projection of the Haar measure. Denote by / the segment 
[^1, 1]. Consider the map 

S:SO(n)-^SO(n-l) xl 

given by the formula (see a notation in the beginning of the section) 

S(.9) = (Ti(5),[g]i) 

Proposition 2.3. The image of the Haar measure dan under the map S is 

const • (1 - x'^Y"-^^/^dxd(Tn-i 



Proof. By SO(n — 1)- equivariance of the map T^ (Proposition 2.2), the 
image of the Haar measure has the form (j){x)dx dan-i- On another side the 
projection of the Haar measure to / is const • (1 — x^)'"^'^)/^ and we obtain 
required form of the function ^(a;) . 

2.4. The map of the orthogonal group to the cub. Consider the maps 

SO(n) ^ SO{n - 1) X [-1, 1] -^ SO(n - 2) x [-1, 1] x [-1, 1] ^ . . . 



As result we obtain the map of the group SO(n) to the cube [—1,1]" ^, given 
by the formula 

(xi, ^2, . . . ,x„_i) = ([T"-2(.g)]i, . . . , [Ti(g)]i, [g]i) 

By Proposition 2.3, the image of the Haar measure under this map is 

n — 1 n — 1 

(i/x(xi, X2, ■ ■ ■ , Xn-i) — const • I I (1 ^ xjp^^'^'''^di^{xi) I I dxj 

Hence for each function / depending on n — 1 variables xi, . . . , a;„-i G [^1) 1] 
we have 

f{[T^'~\9)]i,...,[r\g)]iA9]i)d<yn{g) = 

SO(ri) 

f{xi,X2, . . . , X„_i)d^(xi, 2:2, . . . , Xn-l) (2.1) 

2.5. Multiplicativity. Proposition 2.4. Let g e 0{n). Let m < p ^ n. 
Then 

det(l + [g]p) = det(l + [.9]™) det(l + [T^Cg)]^,™) 

Proof. Let us wright g as block {m+(p—m) + {n—p))x {m+{p—m) + {n—p)) 
matrix: 

/ P Qi Q2 

y i?2 T21 T22 

By the usual formula for determinant of a block matrix, we obtain 

P Oi 



det 
On another side 



Ri Til 



det(l + P) det(l + Til - i?i(l + Py^Qi) 



^,n( , / Tn - i?i(l + Py^Qi T12 -Ri{l + P)-^Q2 

^^> \T2l-R2{l+P)-^Ql T22-R2{l+P)-^Q2 

and now the statement becomes obvious. 

2.6. Some integrals. Theorem 2.5. Assume that the Haar measure of the 
whole group SO(n) is 1. Let Ai, . . . , A„ G C, A„+i = 0, and Re Afc > — (n— fc)/2 
for all k = 0,1, ... ,n. Then 



.. n— 1 n— 1 

/ n det(l + [.g]fe)^'=^^'=+^ da^ig) = TT 

JsO(n) t , t ^ 



T{n-k)T(\k + {n-k)/2) 
(„) ,i-\ "^"^ ' ""' ^"""' llr((n-fc)/2)r(Afc+n-fc) 



(2.2) 



Proof. Let us apply formula (2.1) and Propositfon 2.4. Then our integral 
splits to the product of integrals having the form 

J\{1 - x2)("-fc-2)/2(i „ j:)Xkdx = (2.3) 

_9Afc+n-3R/\ n-fc n-k \ _ oAfc+K-fc-3 r(At + (n-fc)/2)r((7t-fc)/2) 

— Z £>\^Afe+2'2/^^ r(Afc+ri-fe) 

2.7. Some integrals over unitary and symplectic groups. The fol- 
lowing integrals can be evaluated in the same way, the only the obvious integral 
(2.3) is replaced by more complicated integrals over ball \x\ < 1 in C or in H. 

^ -rV r(n-fc + l)r(n-fc+l + Afc+Mfc) 

£i r{n-k + l + Xk)T{n - fc + 1 + /ife) ^ ■ ^ 



/U(n) fc^i 



JSp{n) f.^^ 



^ yr r(2(n - fc + l))r(2(n - fc + 1) + Afc + 1) 

£i^ r(2(n - fc + 1) + Afc/2)r(2(n - fc + 1) + Afc/2 + 1) ^ ' ' 

(we define the compact symplectic group Sp(n) as the group of unitary oper- 
ators in n-dimensional quaternionic space H"; a quaternionic operator A can 
be considered as an operator Ar : R^" — > R^"; we define a determinant of the 
quaternionic operator A as y^detA^). 

2.8. Other ways of evaluation of integrals (2.2), (2.4), (2.5). We 
obtain a verbal evaluation of these integrals, nevertheless our way is not quite 
usual. There exist another way of evaluation of the same integrals. The Cay ley 
transform reduces (2.2) to the form 

/n 
n det(l - [TlkT" dT (2.6) 

fc=2 

where integration is given over the space of all skew-symmetric matrices. Last 
integral can be easily evaluated by the method described in pl[ (in fact (2.6) is 



simpler than integrals from |21 ). Similar way is valid for (2. 4), (2. 5). 



Integral (2.4) reduces to the form 

/n 
H det(l + [r]fc)^- det(l - [T]kr'' dT 
1 1 



fc=i 



where integration is given over the space of all anti-hermitian matrices (T = 
—T*). In this case where are two additional possibilities: we can use Laplace 



transform or we can use Gindikin method of separation of variables (see B, 
chapter 7). 

Remark. If Ai = A2 = • • • = A„, then integral (2.2) is equivalent to one of 
Hua Loo Keng integrals [Q , chapter 2 

§3. Operators of restriction to the boundary 



In this section we assume p < q. 

3.1. Operators of restriction to M^. Denote by 7i° the subspace in 
i?°(Bp^g) consisting of functions analytical in some neighborhood of closed ma- 
trix ball Bp.q. 

Lemma 3.1. The subspace 7i° is dense in H°. 

Proof. There exists a basis in H° consisting of orthogonal homogeneous 
polynomials (see for instance P3|, §1). These functions are eigenfunctions of the 
operator Ai;f{z) = /((I — e)z). The eigenvalues are (1 — e)'^. Hence for each 
£ > the operator A^ is a continuous operator H° -^ i?°. Obviously the image 
of the operator A^ is dense in H° and also it is contained in H° . ■ 

By Ll^^{Mr) we denote the space of locally integrable functions on manifold 
Mr (see Subsection 1.1; if r ^ then Mr is not compact). 

The group 0{p,q) acts in the space Ll^^{Mr) by same formula (1.1), as in 
iJ°(Bp_g) (we only have to think that a matrix z is an element of M^). 

By Jr we denote the operator of restriction of a function / G 7i° to the 
submanifold M^. 

Theorem 3.2. Let a < {q — p + 2r)/2. Then the operator Jr extends to a 
continuous operator i?° -^ Ll^^{Mr). 

Remark. Let a 7^ (if a = then the space _ff° is one-dimensional). In 
general, functions / e iJ°, are discontinuous in point of boundary of the matrix 
ball. In other words, for points a £ ^p,q \ ^p,q the lineal functional / i— j /(a) 
on H° is not well-defined. Hence existence of the restriction operator is not 
obvious. 

Evidently, the operator Jr : il° -^ Ll^^{Mr) is 0(p, (7)-intertwining. Hence 
its kernel ker Jr is a subrepresentation. Hence the orthogonal complement 
(ker Jr)^ also is a subrepresentation. The operator Jr is a bijection from 
(ker Jr)"*" to the image im J^. of the operator J^. Hence we constructed a sub- 
representation in i?°, and this subrepresentation has a natural realization in a 
space of functions on M^. 

3.2. Proof of the theorem 3.2. In [||| (theorems 2.2-2.2) there was 
obtained one general statement on boundary values of holomorphic functions. 
In our case, it implies the following statement 

Theorem 3.3. Let M be a compact subset in the boundary of the matrix 
ball Bp_g. Let pL be a measure supported in M . Assume that 



i) For almost all (with respect to the measure ^ x fj.) points (z, u) e M x M 
there exists a limit as e tends to from the right 

L{z,u)= lim det(l - (1 - e)zM*)"" 

£— > + 

ii) This limit is dominated, i.e. there exists a function h G L^{^ x ^) such 
that 

det(l-(l-£)zM*)^" ^ h{z,u) 

Then the restriction operator of a function f € i?° to the subset M is a 
continuous operator if° — > L^(M). 

First we will explain briefly a way of application of the theorem in our 
case. We want to apply it to the orbit M^. Obviously the limit exists. In fact 
it is sufficient to check the convergence of integral of the function L{z, u) = 
det(l — zu*)~". Nevertheless I don't see a natural measure on M^. Hence there 
is a small hope to evaluate integral if we don't know natural integrand. 

An estimation of convergence also seems heavy since the singularity of the in- 
tegral is very unpleasant. In fact we construct non-directly a calculable integral 
with given singularities, w 

To apply Theorem 3.3, we consider arbitrary compact subset M on the 
manifold M^, we also assume that M is a closure of its interior. Let /i be the 
surface Lebesgue measure on M. It is sufficient to check that the integral 

det{l - zu*)~°'dii{z)dfj,{u) (3.1) 

M JM 

is convergent. The condition i) is a consequence of the convergence. Condition 
ii) follows from the obvious inequality (Q,3.8): 

det(l - c^zu*)^" s; 2P" det(l - zu*y°'. 

It is sufficient to check that the integral 

det(l - zu^y^d^iiz) (3.2) 

M 

is convergent for any u and is dominated by a constant that don't depend on 
u. As it will be shown below, it is sufficient to prove convergence of the integral 
for some u. 

First we consider u = u^^' given by block ((p— r) + r) x {{p — r) + {q—p + r)) 

matrix , n ] ■ Then integral (3.2) transforms to the form 



det[l + z]-Xd^I{z) (3.3) 

M 



Lemma 3.4. Integral (3.3) converges. 

.911 .912 



Lemma 3.5. Let us represent g G SO{q+r) as block matrix g 

V 521 522 

having the size (p + {q + r — p)) x (q + r). Then gn G M^. Conversely each 
matrices z G M^ is a left upper corner of some matrix g G SO{q + r). 

Proof of Lemma 3.5 is obvious. 

Proof of Lemma 3.4. By Lemma 3.5 the map t : g t-^ gu takes SO(gr + r) 
to Mr- Obviously, the image of the Haar measure on SO{q + r) under the map 
T is a measure equivalent to a surface Lebesgue measure on M,.. Hence it is 
sufficient to check convergence of the integral 

Act[l + g\p"^.daq+r{g) 

SO(q+r) 

But this integral was evaluated in §2 ( we substitute Ai = • • • = Ap_r-i = 
—a; \p-r = • • • = to formula (2.2)). ■ 

Lemma 3.6. The kernel dct(l — zm*)^" satisfies to the equality 

det(l - z[»1(u[»1)*)-" = dct(l - zu')-" dct(a + zc)" det(a + zm)" (3.4) 

Proof is obvious. 

Consider a compact subset i C 0(p, q) such that for each w G M there exists 
gv € L satisfying the condition -yl^"' — u'-^K Denote by K the set of all points 
in Mr having the form m^' where u £ M and g G L. Obviously, K is compact. 

Substitute w = z'^l to the integral (3.2). By formula (3.4), we obtain 

det(l - u'-°^w*)-"H{w, u) dfi{w), (3.5) 

where M^^"' denote the image of the set M under the map g^, and H{w, u) is 
some function (a product of Jacobian and additional factor which appears from 
(3.4)). Obviously, the function H(w,u) is bounded by some constant C. Hence 
integral (3.5) is dominated by quantity 

C f det(l - u'^°^w')-°'diJ,{w) 

JK 

This quantity doesn't depend on u and it is finite by Lemma 3.4. Theorem is 
proved. 

3.3. Restriction of partial derivatives to the boundary. In Subsec- 
tion 3.2 we restricted functions to M^. Also it is possible to restrict partial 
derivatives of the function / G iJ°. The existence of restriction operator for 
partial derivative -^— depends on convergence of integral 



M J M 



dzij duij 



det(l - zu*)-" 



dz du, 



10 



where M C M^ is a compact subset. The integrand has the form |P(z, m)| det(l— 
ZM*)~"^^ where P{z,u) is a polynomial. Hence we can apply the same argu- 
ments. As a result, we obtain the following theorem. 

Theorem 3.7. Let a + 2k < {q — p + 2r)/2. Let d" he a partial derivative 
of order k. Then the operator of restriction of a function d'^ f{z) to M^ is a 
continuous operator from _ff° to ^^^^(Mj,). 

Now we will slightly formalize the picture. 

Denote by Af{Mr) the conormal bundle to M^ in R?"?. By S''Af{Mr) we 
denote fc-th symmetric power of the conormal bundle. By JF^ we denote the 
set of all / G 7i° whose partial derivatives of order ^ k vanish on M^. Then 
elements of quotient space !F^~^ /T^ are identified with sections of the bundle 

Theorem 3.7 implies the following statement: 

Let a + 2k < {q — p + 2r)/2. Then the operator of restriction of partial 
derivatives of order k to M^ is a well-defined operator from the space iJ° to the 
space of L^^^- sections of the bundle S'^A/'(Mr).n 

3.4. Some comments. Thus we obtain the intertwining operator Jr^k 
from _ff° to the space of sections of bundle S^N'{Mr). Its image imJr^k is a 
quotient space of a Hilbert space. Hence imJ^^fc has a natural structure of a 
Hilbert space, (we emphasis that the Hilbert topology in imJ^./c is stronger 
than topology induced from Ll^^). We denote the representation of the group 
0{p,q) in imJr,k by 

S?,fc = S?,fe(p,g) (3.6) 

The quotient space of a Hilbert space H by arbitrary closed subspace L can 
be canonically identified with the orthogonal complement L-^ to L. Hence the 
representation S"^ is a subrepresentation in the kernel-representation Ta- 

Hence we obtained a canonical family of subspaces (3.6) in iJ°(Bp^q), we 
name these subspaces by the term blocks. 

If a is sufficiently large (i.e. a > {p + q)/2 — 1), then the operators of 
restriction to submanifolds M^ are not defined. In this case blocks S"j,(p, g) 
don't exist. It is known that for a > {p + q)/2 — 1 the representation of 0{p, q) 
in H°{Iip^g) is equivalent to the representation of 0{p, q) v in L^ on symmetric 
space Bp qQ 

For all a > p — 1 the summand ("block"), that equivalent to L'^{Bp g), is 
present in decomposition of the kernel-representation. 

For a < {p + q)/2 — 2 the block S"]^ q appears. This summand corresponds 
to the operator of restriction of function to the maximal boundary orbit Mp„i. 

* the algebraic version of this construction arises to |13[ 

^this statement for arbitrary scalar- valued kernel-representations is proved in cite |24| ; 
earlier similar statement for kernel representations of some groups were obtained (in different 
degree of generality) in 1), [Q, Q). 
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For a < {p + q)/2 — 3 block '^p-2fi appears. After point a = [p + q)/2 — 4 
two additional blocks Sp_;^ ;^ and Sp_3 g appears etc. 

At the point a = p—1 (it is the end of continuous series, see 1.2) the largest 
block L^{Qp,q) disappears from decomposition. Also at this point all blocks 

Spl]^ 2i '='p~i,2i ■^p~i,3 ^t*^- disappear. But the block S^Ij^ q at this point still 
alive. 

If we continue decreasing of a (after point a = p — 1 it is a discrete process, 
see Subsection 1.2) "new" blocks continue their appearance. Nevertheless "old" 
blocks quickly disappear (this process can be checked using Plancherel formula, 
see Addendum B). 

At the point a = only the block Sq q is still alive, and this block is one- 
dimensional representation. 

Let us briefly discuss spectra of representations S"^. 

Proposition 3.8. Each subrepresentation in the kernel-representation Ta 
contains an 0(j>) x 0{q) -invariant vector. 

Proof. The function f{z) = 1 is an 0(p, g)-cyclic vector in iJ° (since its 
orbit is exactly super-complete basis). Hence the projection of the function 
f{z) = 1 to any invariant subspace is a cyclic vector in this subspace. Hence 
the projection is non zero. 

Proposition 3.9. a) The representation of the group 0{p,q) in Sqq is 
irreducible 

b) The representation oJO{p,q) in Sq j, is a finite sum oj irreducible repre- 
sentations. 

Proof, a) Indeed the constant is the unique 0{p) x 0(g)-invariant function 
on Mq. 

b) We have to check that the space of 0{p) x 0((7)-invariant sections of the 
bundle S'^A/'(Mo) is finite-dimensional. But dimension of this space can't exceed 
the dimension of fiber. 

Remark. "Block" Sq q which corresponds to operator of restriction to the 
compact orbit Mq was discovered in the paper of G.I.Olshanskii and myself 
[E3|, §3 (under more rigid conditions to parameters of representations) a < 
(g-2p-M)/2. 

Remark. Spectrum in blocks ^"^ (where < r < p) is purely continuous. 
It consist of representations induced from unitary representations of groups 
0{p—r,q — r). The last representations are contained in spectrum of Sp J^J^'^ (p — 
r,q-r). 

Remark. "Additional" block with continuous spectrum in kernel-representations 
was discovered earlier only for the case G = U(2, 2) in the paper [Q. In [Q 
I conjectured that the spectrum of kernel-representations for small values of a 
contains many various series. The proof of this conjecture is the subject of this 
paper. 
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§4. General kernel-representations. 



4.1. Definition. By pA we denote finite dimensional holomorphic repre- 
sentation of the group GL(n, C) defined by a signature A — (Ai, . . . , A„), where 
Ai ^ . . . ^ A„. By Va we denote the representation pA- Recall that there exists 
a canonical U(n)-invariant scalar product in Va- 

Consider the representations pA', PA" of the groups GL(p, C), GL((7, C) de- 
fined by signatures A' = (A'^, . . . , A^), A" — (A", . . . , A^') respectively. We will 

use notations A = (A'^, . . . , A^; A", . . . , A^') and also V~^ — Va' ® Vk"- 
Consider the function 



Kj^i^z, u) — pA'(l ~ •2:-u*) (g) pA"(l — wz*); z, u e B, 



■p,« 



taking values in the space End(V^) of linear operators in V^. 

Assume that K-^{z, u) is a matrix-valued positive defined kernel. Recall that 
this means positive definitcness of the kernel 

on the space X = Bp g x l^ (for the condition of positive definiteness of the 
kernel A(z, u) we obtained in ||2^, see also any text on classification of highest 
weight unitary representations, see also ||23|| . 

The kernel L {{z, ^); {u, 77)) defines a Hilbert space consisting of functions on 
the space X — Bpq x V^. But these functions are linear in the variable ^ € V^. 
Hence we can consider them as Vr -valued functions on Bp q. 

As a result we obtain the space H- , which consists of V^-valued functions on 
Bp q. The group 0(p, q) acts on H~ by unitary operators given by the formula 



T~A9)f{z) = PA' (a + zc) ® pA"(-d + czlsl) 



/(^ 



[9]^ 



4.2. Operators of restriction to the boundary. Theorem 4.1. a) 
Assume A" — A' < {q — p + 2r)/2. Then the operator of restriction to an orbit 
Mr is a well-defined operator in H~ . 

b) Assume 2k + X'(~X < {q — p + 2r)/2. Then the operator of restriction of 
partial derivatives of order k to the orbit Mr is a well-defined operator in H° . 

Proof is similar to the proof of Theorem 3.2, it is necessary also use simple 
considerations from |gj,4.5. 

4.3. Comments. As above, we obtain decomposition of kernel-representation 
to "blocks" S^^,, where r is a parameter enumerating orbits, and k is an order 
of partial derivatives. 

Existence of operator of restriction to Mq was discovered in pq|,§4 (under 
more rigid restriction to the signature A). In particular this construction gives 
a simple way for obtaining of "exotic" unitary representations of the groups 
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0{p,q) (see [p3[ ,§§4-5). Emphasis also that this representations admit a Hmit 
as g -> oo (]23|, §6). 

The representations of 0(p,q) in Sqq are reducible; their decomposition is 
discussed in [g3| , §5. As above, the representation of 0{p, q) in all spaces Sq j. 
has a purely discrete spectrum. Nevertheless the spectrum of blocks S^^ for 
r > is still sophisticated and contains representations of different types. Here 
I don't have likely conjectures. 

Addendum A. Hermitization of symmetric spaces. 

Consider the following list. 

1. G = GL(n, M) G° = Sp(2n, M) 

2.G = 0{p,q) G°=V{p,q) 

3. G = Sp(2n,R) G° = Sp(2n,]R) x Sp(2n,R) 

4. G = GL(n,C) G° =U(n,n) 

5. G = SO(n,C) G° = S0*(2n) 

6. G - Sp(2n, C) G° = Sp(4n, R) 

7. G = U(p,q) G°=U(p,g)xU(p,g) 

8. G = GL(n, H) G° = S0*(2n) 

9. G^Sp{p,q) G° =U(2p,2g) 
10.G==SO*(2n) G° =S0*(2n) X S0*(2n) 
ll.G = S0(2,n) G° =SO(2,n)xSO(2,n) 
12. G = SO(l,p) X S0(1, q) G° = S0(2,p + q) 

Denote by K, K° the maximal compact subgroup in G, G° respectively. 
Then G° /K° is a hermitian symmetric space and G/K C G° / K° is a totally 
real subspace, and 

dimu G/K = dime G°/K° 

We say that G° /K° is hermitization of G/K. If G/i^ is hermitian then its 
hermitization is G/K x G/K. We see that all classical riemann symmetric 
spaces have hermitization|j 

We define kernel-representation as restriction of an unitary highest weight 
representation^ p of the group G° to the subgroup G^ 

^Unfortunately I don't know the author of this observation. First time I heard this 20 
years ago. 

^A highest weight representation of the group G° is a representation in a space of holo- 
morphic vector- valued functions on G° / K° 

^Preferences: fl] (the casesG = U(p, g), Sp(2n,M), SO*{2n), SO{2, n)), |23| (G = 
0(p,g), U(p,g), Sp(p, q)), |24| , fcoj] (some discussion of other cases); there exists an idea to 
extend a definition of Isernel representation to pseudoriemann symmetric spaces, see |4l. 
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Addendum B. Analytic continuation of the Plancherel 

formula 

B.l. Plancherel formula. Plancherel formula for all classical series 1-10 
of kernel-representations for large values of the parameter a there was obtained 
in p^ (this method is valid also for future tubes). F] In all cases the Plancherel 
measure has the form 

V{a)T\\r{-{a~h + isk))f——^-— where Sk.eR (B.l) 

fc=i \ ) \ J 

Here r is the rank of the group; isi, is2T--,iSr are the standard parameters of 
spherical functions, spherical functions of unitary representations of non degen- 
erate principal series corresponds to real Sfe; f./^sM-is) '^^ Gindikin-Karpelevich 
measure (see ||ll|), and V{a) is a holomorphic factor. The parameter h is the 
least value of a for which the kernel-representation is square integrable. 

In this formula, the Plancherel measure is supported on unitary principal 
series. We have seen that for small a spectrum of kernel-representation is com- 
plicated and hence Plancherel formula (B.l) is not correct for small a. 

B.2. Analytic continuation of the Plancherel formula. Plancherel 
formula for a < h can be obtained by the analytic continuation of formula 
(B.l). We will give the final formula for the cases G — 0{p,q), lJ{p,q) (for 
other cases see p2[). 

Denote by gt an element of 0{p,q) having eigenvalues e*''=. Let 9o be the 
same as in Subsection 1.2. Then 



<T^{gt)Oo,0o >= ijcosh-" tk 



k=l 

Plancherel formula is equivalent to expansion of matrix element < Ta0o,9o > 
by spherical functions of 0{p,q). Let us denote spherical functions of 0(p, g) 
by ^isi,...,isr- By *I'jsi,...,is,.(ii, ^2, ■ • ■ tp) we denote a value of spherical fmiction 
on gt. 

^For the groups U{p,q), Sp(2n,R), SO*(2n), SO(n,2) and large values of parameter a 
the Plancherel formula was obtained by Berezin in 1978 (|H1), Shortly after Berezin died and 
proof was not published until M]. For the groups 0(p, 1), Sp(p, 1) the Plancherel formula 
was obtained in B 
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In these notations the Planchcrcl formula for a > {p + q)/2 — 1 has the form 

p 
IT cosh^"tfe = 

-^■n r(. -k + 1) L. n r((.4(.+,)+i+.s.)/2)r((.-l 
fc=i ^ ' •^^<='* fc=i 

n 



-2(P+9) + l-*5fc)/2)x 



^ , , r((g-p)/2 + »gfc)r((g-p)/2-».fc) ^^ 
r{isk)T{-isk) 



fe=i 
n ('*^ ~ sj*)tanh-(sfe - s/)tanh-(sfe + sj) x 

X $isi,...,is,(il,i2,---,ir)c?Sl...rfSp (B.2) 

where ^ is a constant. 

Now we will Wright the analytic continuation of formula (B.2). Let r = 
0, 1, . . . ,p. Let Ml, 1*2, ■ ■ ■ ,Ur be nonnegativc integers. Let 

Wj = Ml + • • • + Uj + j/2 

Then 



[] cosh-"(i„) = 



?n— 1 



r— 111 ,...,u^:'u;i,</i — a 

X / QM(Q;|Sr+l7 • • • 7 'Sp)$Q,_(p^q-)/2+2«)i,...,Q-(p+(j)/2+2tUp,iSr+i,...,iSp (^li • • • i ^p) X 

dsr+1 . . . dsp (B.3) 
where 

2P-'-p!(27r)'' (-1)"^+-+"- yr Ti^+Wm-Wk) 



(p-r)! n^j! .i;„< r(M;„-M;fc)(^ + w^fc-i -w„)„j^ 



and A is a constant, 
1 



K(a) 



r(a -P+1 + +2wk)T{-a + q-l- 2wk) 

k 



n 



n 



•^ r(-a + i(p + (?) - 2u;fe)(Q; - \{p + q) + Wk + Wfe-l + \)u,, 

r(i - a + i(p + q) - M7fc - W.m) 



r(-a+ i(p + (7) - Wfe -M;„)(a- |(p + g) + Wm + w^-i + i)„j^ 
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and 



Quia\s, 



r+l: • ■ • 7 ^p) 



P I 

Yl |r(-(a-(p + (z)/2 + l + zs„)+u;„_i)|2x 



n— r+1 



r(i(l - a + i(p + g) - 2wk + isi) 



ki'rj>r ^(hi-O" + Up + 9) - 2wk + isi)){^{a - i(p + q) + Wk+ isi)u^ 



n (^i^ ^ *») tanh(-(sj - s„)) tanh(-(s, + s„)) 



/>n>r 

In the case G — U(p, (7) we obtain 



\\ cosh""(t„ 



E E G»-^»(«) 



r— wi,...,i(;^ 

X / Qw\'^\Sr+l, ■ ■ ■ , 'Sp)$Q,_(p+g)_|_i_|_2toi,...,Q-(p+(2) + l+2M)r,iSr+i,...,iSp(ilj • • • i ^p) X 

(isr+1 . . . dsp (B.4) 
where 



(-^?« — A ■ 



(p-r)! 



n^^ni- 



k=l 



Wk 



Wl) 



k<l^r 



^ is a constant, 

P 1 



n 



j^ll^ -, -^—, -, — — X 



■^ r(a/2 - m + 1)2 ^^1 r(-a + (p + g) - 1 + 2wk){a - {p + q) + 1 + Wk)w^ 



Yl ia-ip + q) + l + Wk + 



Wl) 



k<l^r 



17 



and 

Qw{a\Sr+l, ■ 



n 

n— r+1 



|r(i((g - p + 1 + zs„))Hr(i(a ^p-q+l + t.s„)\^ 



\r{isr, 



n (^ 



n>m>r 



' ^)' 



B.3. Negative integer a. Analytic formula (B.3),(B.4) are valid for all 
values of the parameter a G M. Nevertheless it has visible sense only for 

a = 0, 1, 2, . . . ,p — 1 or a>p—l 

when representation T^ is unitary, and for 

a = 0,-1,-2,... 

when the representation T^ is finite-dimensional. In the last case the right parts 
of identities contain only finite number of summands with r = p { F-factors in 
denominator annihilate all summands with r < p). In the case G = \J{p,p) 
we obtain Pickrell decomposition , [Q. In this place our work also has an 
intersection with a recent preprint of G.Zhang [p3[ . 

For over values of parameter a Plancherel formula (B.3), (B.4) are elements 
of mysterious nonunitary harmonic analysis (see, for instance |15[,|p[|). 
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